Motion sensor apparatus having a plurality of light sources

ABSTRACT

A motion sensor device according to an embodiment of the present disclosure includes an image sensor ( 101 ) and LED light sources ( 102, 103, 104 ). Using three images captured by the image sensor ( 101 ) when only the LED light source ( 102 ) is turned ON, when only the LED light source ( 103 ) is turned ON, and when only the LED light source ( 104 ) is turned ON, respectively, the motion sensor device estimates the distance to an object based on the luminance ratio of the object.

TECHNICAL FIELD

The present application relates to a motion sensor device with multiplelight sources.

BACKGROUND ART

Patent Document No. 1 discloses a technique for measuring the distanceto an object (or target) which is either standing still or moving basedon a plurality of images that have been captured by a single imagesensor by projecting light time-sequentially from multiple light sourcesonto the object.

Non-Patent Document No. 1 discloses a gesture interface system fordetecting a human hand's motion with a near-infrared ray, and says thata human being's skin exhibits a higher reflectance to an infrared raywith a wavelength of 870 nm than to an infrared ray with a wavelength of970 nm.

CITATION LIST Patent Literature

-   Patent Document No. 1: Japanese Laid-Open Patent Publication No.    2001-12909

Non-Patent Literature

-   Non-Patent Document No. 1: Mariko Takeuchi, Kunihito Kato and    Kazuhiko Yamamoto, “Establishing Gesture Interface System Using    Near-Infrared Skin Detection”, the Institute of Image, Information    and Television Engineers (ITE) Technical Report Vol. 34, No. 34, ME    2010-122 (August 2010)

SUMMARY OF INVENTION Technical Problem

According to the conventional technologies, the accuracy or sensitivityof the distance measured may decline depending on the position of theobject. Thus, an embodiment of the present disclosure provides a novelmotion sensor device which can overcome such a problem with theconventional technologies.

Solution to Problem

A motion sensor device according to an aspect of the present disclosureincludes: an image sensor; first, second and third light sources; and acontroller configured to control the image sensor and the first to thirdlight sources. The controller is configured to: make the image sensorcapture a first frame with light emitted from the first light source ata first time; make the image sensor capture a second frame with lightemitted from the second light source at a second time; make the imagesensor capture a third frame with light emitted from the third lightsource at a third time; generate a first piece of estimated distanceinformation based on a ratio of a luminance of an object obtained from afirst image produced by capturing the first frame to a luminance of theobject obtained from a second image produced by capturing the secondframe; generate a second piece of estimated distance information basedon a ratio of the luminance of the object obtained from the first imageto a luminance of the object obtained from a third image produced bycapturing the third frame; and obtain information about a distance tothe object by either choosing one from, or synthesizing together, thefirst and second pieces of estimated distance information.

A motion sensor device according to another aspect of the presentdisclosure includes: an image sensor which is able to detect light witha first wavelength and light with a second wavelength that is differentfrom the first wavelength; first and second light sources which emitlight with the first wavelength; third and fourth light sources whichemit light with the second wavelength; and a controller which isconfigured to control the image sensor and the first to fourth lightsources. The controller is configured to: make the image sensor capturea first frame with light emitted from the first and fourth light sourcesat a first time; make the image sensor capture a second frame with lightemitted from the second and third light sources at a second time;generate a first piece of estimated distance information based on aratio of a luminance of a first wavelength component of an objectobtained from a first image produced by capturing the first frame to aluminance of the first wavelength component of the object obtained froma second image produced by capturing the second frame; generate a secondpiece of estimated distance information based on a ratio of a luminanceof a second wavelength component of the object obtained from the firstimage to a luminance of the second wavelength component of the objectobtained from the second image; and obtain information about thedistance to the object by either choosing one from, or synthesizingtogether, the first and second pieces of estimated distance information.

A motion sensor device according to still another aspect of the presentdisclosure includes: an image sensor; first, second and third lightsources; and a controller configured to control the image sensor and thefirst to third light sources. The controller is configured to: make theimage sensor capture a first frame with light emitted from the firstlight source at a first time; make the image sensor capture a secondframe with light emitted from the second light source at a second time;and make the image sensor capture a third frame with light emitted fromthe third light source at a third time, and wherein luminances of anobject obtained from first, second and third images produced bycapturing the first, second and third frames are called first, secondand third luminances, respectively, the controller is configured toobtain information about the distance to the object based on a ratio ofa fourth luminance which is obtained by either synthesizing together, orchoosing one from, the first and third luminances to the secondluminance.

A motion sensor device according to yet another aspect of the presentdisclosure includes: an image sensor which is able to detect light witha first wavelength and light with a second wavelength that is differentfrom the first wavelength; first and second light sources which emitlight with the first wavelength; third and fourth light sources whichemit light with the second wavelength; and a controller which isconfigured to control the image sensor and the first to fourth lightsources. The controller is configured to: make the image sensor capturea first frame with light emitted from the first and fourth light sourcesat a first time; and make the image sensor capture a second frame withlight emitted from the second and third light sources at a second time,and wherein luminances of the first wavelength component of the objectobtained from the first and second images produced by capturing thefirst and second frames are called first and second luminances,respectively, and luminances of the second wavelength component of theobject obtained from the first and second images are called third andfourth luminances, respectively, the controller is configured to obtaininformation about the distance to the object based on a ratio of a fifthluminance which is obtained by either choosing one from, or synthesizingtogether, the first and second luminances to a sixth luminance which isobtained by either choosing one from, or synthesizing together, thethird and fourth luminances.

An electronic device according to the present disclosure includes: amotion sensor device according to any of the embodiments describedabove; and a display which changes what to present thereon in responseto an object's motion detected by the motion sensor device.

An integrated circuit according to the present disclosure is to be usedin a motion sensor device according to any of the embodiments describedabove, and includes: a timing controller which is connected to the imagesensor and the light sources to control timings of exposure and lightemission; an extreme value searching section which processes imagecapturing data to search an area with a relatively high luminance; acoordinate memory which stores the coordinates and luminance of the areathat has been searched by the extreme value searching section; and adistance calculating section which calculates estimated distanceinformation based on a luminance ratio by choosing frames that have beenshot in the same area under different conditions from data in thecoordinate memory.

A distance estimating method according to the present disclosure isperformed by a motion sensor device according to any of the embodimentsdescribed above, and includes: searching for an extreme value byextracting selectively a range with a relatively high light intensityfrom data of captured frames and by calculating its coordinates andlight intensity; calculating the ratio of luminances that have beenextracted in the extreme value searching step and that have beenselected from among luminances of frames shot under differentconditions; and converting the luminance ratio calculated in theluminance ratio calculating step and the coordinates searched for in theextreme value searching step into a distance.

A computer program according to the present disclosure is defined tomake a motion sensor device according to any of the embodimentsdescribed above perform the steps of: searching for an extreme value byextracting selectively a range with a relatively high light intensityfrom data of captured frames and by calculating its coordinates andlight intensity; calculating the ratio of luminances that have beenextracted in the extreme value searching step and that have beenselected from among luminances of frames shot under differentconditions; and converting the luminance ratio calculated in theluminance ratio calculating step and the coordinates searched for in theextreme value searching step into a distance.

Advantageous Effects of Invention

A motion sensor device according to an embodiment of the presentdisclosure can prevent errors from being caused in the distance beingmeasured.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A A cross-sectional view schematically illustrating a crosssection of a motion sensor device with two light sources.

FIG. 1B A top view of the device shown in FIG. 1A.

FIG. 2 A graph showing how the relative radiation intensity of an LEDlight source changes with the angle of radiation.

FIG. 3 Illustrates the angle of radiation of an LED light source.

FIG. 4A Illustrates how an object 104 is irradiated with light that hasbeen emitted from a first LED light source 102.

FIG. 4B Illustrates how an object 104 is irradiated with light that hasbeen emitted from a second LED light source 103.

FIG. 5 A graph showing how the luminance of image capturing data changeswith a pixel location on a line.

FIG. 6 A graph showing how the luminance ratio changes with the distanceat a certain angle of radiation.

FIG. 7A Illustrates how an object 104 that has moved slightly isirradiated with the light emitted from the first LED light source 102.

FIG. 7B Illustrates how the object 104 that has moved slightly isirradiated with the light emitted from the second LED light source 103.

FIG. 8 Schematically illustrates a sensitivity range of a motion sensordevice with two light sources.

FIG. 9A Illustrates an arrangement of a light source unit and an imagesensor in a light source unit according to a first embodiment of thepresent disclosure.

FIG. 9B A timing chart showing when the light sources and image sensorare activated in the first embodiment of the present disclosure.

FIG. 10 Illustrates low-sensitivity ranges according to the firstembodiment of the present disclosure.

FIG. 11 Illustrates a configuration for a motion sensor device accordingto the first embodiment of the present disclosure.

FIG. 12 A block diagram illustrating an exemplary configuration for amotion sensor device according to the first embodiment of the presentdisclosure.

FIG. 13 A flowchart showing the procedure of calculating the distanceaccording to the first embodiment of the present disclosure.

FIG. 14A A top view illustrating another exemplary arrangement accordingto the first embodiment of the present disclosure.

FIG. 14B A top view illustrating still another exemplary arrangementaccording to the first embodiment of the present disclosure.

FIG. 15A Illustrates an arrangement of a light source unit and imagesensor according to a second embodiment of the present disclosure.

FIG. 15B A timing chart showing when the light sources and image sensorare activated in the second embodiment of the present disclosure.

FIG. 16 Illustrates low-sensitivity ranges according to the secondembodiment of the present disclosure.

FIG. 17 Shows the sensitivity characteristics of an image sensoraccording to the second embodiment of the present disclosure.

FIG. 18 Illustrates an exemplary product in which a motion sensor deviceaccording to the first embodiment of the present disclosure is built.

DESCRIPTION OF EMBODIMENTS

The basic principle on which the distance to an object (or subject) canbe measured by a motion sensor device according to the presentdisclosure will be described.

First of all, look at FIGS. 1A and 1B. FIG. 1A is a cross-sectional viewschematically illustrating a cross section of a motion sensor device,and FIG. 1B is a top view of the device shown in FIG. 1A.

The device shown in FIGS. 1A and 1B includes an image sensor 101 whichis arranged at its center and two LED light sources 102 and 103 whichare arranged on the right- and left-hand sides of the image sensor 101.In the example illustrated in FIGS. 1A and 1B, the image sensor 101 andLED light sources 102, 103 are mounted on a single substrate 100. Theimage sensor is a solid-state image sensor in which a huge number ofvery small photosensitive cells (photodiodes) are arranged in columnsand rows, and is typically a CCD (charge-coupled device) type or a CMOStype.

In FIG. 1A, illustrated schematically are light 102 a emitted from thefirst light source 102 and light 103 a emitted from the second lightsource 103. This device can measure the distance to an object ofmeasurement (i.e., target) by capturing an image with the LED lightsources 102, 103 turned ON alternately. It should be noted that “tomeasure the distance” will also refer herein to calculating an estimateddistance from the image sensor to the object or obtaining an estimatedvalue indicating the object's position in a space. Examples of theobjects include a human being's hand(s) or finger(s) and a pen orsomething else held in his or her hand. The object may be in motion. Athree-dimensional motion sensor device which can obtain in real timeeither the distance to a person's fingertip that is moving at highspeeds or an estimated value indicating the fingertip's position may beused as an “input device” in various kinds of electronic devicesincluding computers, tablet terminals, smartphones, game consoles andconsumer electronic devices.

FIG. 2 is a graph showing the radiation pattern (i.e., the lightdistribution characteristic) of the light emitted from each of the LEDlight sources 102 and 103. The abscissa of this graph represents theangle θ defined by the radiation direction with respect to a normal N tothe substrate 100 as shown in FIG. 3. On the other hand, the ordinate ofthis graph represents the relative radiation intensity. In the followingdescription, the angle θ defined by the radiation will be sometimeshereinafter referred to as the “angle of radiation”. It should be notedthat the relative radiation intensity value corresponds to theilluminance of an object which is arranged at a position in a directionthat defines a particular angle with respect to the light source (i.e.,radiation illuminance).

As can be seen from FIG. 2, the radiation emitted from each of the LEDlight sources 102 and 103 exhibits the highest intensity when the angleθ is zero degrees. In the example shown in FIG. 2, the LED light sources102 and 103 have a light distribution characteristic, of which theradiation intensity can be approximated by I₀×cos θ. However, the LEDlight sources 102 and 103 do not have to have the light distributioncharacteristic shown in FIG. 2. In addition, the radiation emitted fromthe LED light sources 102 and 103 does not have to be visible light butmay also be an electromagnetic wave such as an infrared ray which fallswithin a wavelength range to which the human vision is insensitive. Inthis description, the radiation emitted from a light source will besometimes simply referred to as “light” for the sake of simplicity. Thisterm “light” does not have to be visible light but may also refer to anyof various kinds of electromagnetic waves which can be detected by theimage sensor.

Next, it will be described how the device described above measures thedistance to the object.

First of all, look at FIGS. 4A and 4B. FIG. 4A illustrates how theobject 104 is irradiated with light that has been emitted from the firstLED light source 102 and how part of the light reflected from the object104 is incident on the image sensor 101. On the other hand, FIG. 4Billustrates how the object 104 is irradiated with light that has beenemitted from the second LED light source 103 and how part of the lightreflected from the object 104 is incident on the image sensor 101. Theobject 104 is supposed to be located at substantially the same positionin both of FIGS. 4A and 4B.

At a first time, this device gets a first shooting session done by theimage sensor 101 with the LED light source 102 turned ON and the LEDlight source 103 turned OFF as shown in FIG. 4A. Next, at a second time,the device gets a second shooting session done by the image sensor 101with the LED light source 103 turned ON and the LED light source 103turned OFF as shown in FIG. 4B. The durations (i.e., exposure times) ofthe first and second shooting sessions are supposed to be short enoughto be able to handle the object 104 as substantially a still object.

When the first shooting session is carried out, part of the lightemitted from the LED light source 102 is reflected from the object 104and incident on the image sensor 101. As a result, a luminance imagecorresponding to the intensity of the light incident on the image sensor101 is obtained. In the same way, when the second shooting session iscarried out, part of the light emitted from the LED light source 103 isreflected from the object 104 and incident on the image sensor 101. As aresult, a luminance image corresponding to the intensity of the lightincident on the image sensor 101 is obtained.

The object's (104) luminance (which is either its luminance distributionor luminance image) can be obtained based on the two image framescaptured as a result of the first and second shooting sessions. In thisdescription, the “luminance” does not refer herein to a psychophysicalquantity with the unit [candela/m²] but refers herein to a “relativeluminance” to be determined for each pixel of the image sensor andcorresponds to the quantity of light or quantity of radiation. Each ofthe pixels that form each image frame has a “luminance value”corresponding to the quantity of light that the pixel has received.

Since the object 104 has its own size, each image representing theobject 104 is usually comprised of multiple pixels. The “luminance” ofthe object 104 can be determined by various methods based on theluminance values of those pixels that form the object (104) image. Forexample, the luminance of the brightest “pixel” or “pixel block” of theobject (104) image may be regarded as the luminance of the object 104.Or the average luminance of all pixels that form the object (104) imagemay be regarded as the luminance of the object 104.

FIG. 5 is a graph showing the luminance value of a single horizontalline that runs across the object (104) image in each of the two imageframes that have been obtained by the method described above. Theabscissa indicates the location of a pixel on a particular horizontalline in the image, and the ordinate indicates the luminance. In thisgraph, the curve 301 represents the luminance when the LED light source102 is ON, and the curve 302 represents the luminance when the LED lightsource 103 is ON.

In the example shown in FIG. 5, each of the curves 301 and 302 has asingle peak. Specifically, the curve 301 has an extreme value 303 at acertain pixel location, and the curve 302 has an extreme value 304 atanother pixel location. The horizontal interval between the respectivecoordinates of the extreme values 303 and 304 of the curves 301 and 302is indicated by the width 305.

As described above, the object 104 is substantially standing stillbetween the two frames. Thus, the difference is made between the curves301 and 302 because the radiation produced by the LED light source 102has a different pattern from the radiation produced by the LED lightsource 103. The ratio of the luminance of the image captured by makingthe light emitted from the LED light source 102 and then reflected fromthe object 104 be incident on the image sensor 101 to that of the imagecaptured by making the light emitted from the LED light source 103 andthen reflected from the object 104 be incident on the image sensor 101depends on the relation between the distance from the LED light source102 to the object 104 and the distance from the LED light source 103 tothe object 104.

The distance to the object can be measured based on the ratio of theluminances of the images captured. FIG. 6 is a graph showing anexemplary relation between the distance and the luminance ratio in adirection which defines an angle of 45 degrees with respect to the imagesensor 101. In the graph shown in FIG. 6, the abscissa indicates therelative distance to the object and the ordinate indicates the luminanceratio in a situation where LED light sources with the characteristicshown in FIG. 2 are arranged on the right- and left-hand sides at apredetermined distance from the image sensor 101. The “distance” on theaxis of abscissas is measured based on the distance between the imagesensor 101 and the LED light source, and a distance of “1” is equal tothe distance between the image sensor 101 and the LED light source.

The object's luminance (or illuminance) attenuates inverselyproportionally to the square of the distance from the LED light sourceto the object. Thus, the luminance ratio varies according to thedistance. Since the radiation characteristic shown in FIG. 2 is alreadyknown, the distance can be detected or estimated accurately based onthis radiation characteristic.

FIG. 6 shows an exemplary relation between the distance and theluminance ratio when the radiation angle θ is 45 degrees. The relationsbetween the distance and the luminance ratio can be obtained in advancein the same way with respect to multiple different angles, too. Theobject's angle can be obtained based on the position of the object to becaptured by the image sensor.

As can be seen from FIG. 6, if the distance between the object and theimage sensor is longer than approximately one, the distance can bemeasured based on the ratio of the extreme values 303 and 304.

In the example described above, light sources, of which the relativeradiation intensity changes with the radiation angle, are used. However,this measuring method can also be adopted even when light sources thatdo not have such a characteristic are used. Unless light sources whichemit parallel light rays are used, the intensity of the light shouldhave some light distribution characteristic in a three-dimensionalspace. That is why such light sources can also be used to measure thedistance. For example, even in “point light sources” of which the lightdistributions are isotropic, the illuminance and luminance on the objectalso attenuate inversely proportionally to the square of the distancefrom the light sources. Thus, even such light sources can also be saidto be light sources having different radiation patterns in athree-dimensional space.

Next, look at FIGS. 7A and 7B, which illustrate how a shooting sessionis performed on the object 104 that has moved from the position shown inFIGS. 4A and 4B. As long as an image can be captured and the distancecan be estimated quickly, even the distance to a moving object can alsobe measured by the method described above. By illuminating the objectwith the light sources 102 and 103 alternately and capturing the objectimage using the image sensor 101 repeatedly a number of times, theposition of the moving object 104 can be detected. As a result, thechange in the position of the object 104, or its motion, can bedetected.

The present inventors discovered that in a range where the distancesfrom the two LED light sources 102 and 103 to the object 104 were equalto each other, the device described above could measure the distanceless accurately. Such a range will be hereinafter referred to as a “lowsensitivity range”. If the distance on the axis of abscissas in thegraph shown in FIG. 6 is equal to or smaller than one, the shorter thedistance, the higher the luminance ratio gets. That is why the decisioncannot be made, just by the luminance ratio, whether or not the objectis located at a “close range” where the distance to the object is equalto or smaller than one.

FIG. 8 schematically illustrates low-sensitivity ranges for the devicedescribed above. In FIG. 8, illustrated are a low-sensitivity range 504to be produced when the distance is too short and a low-sensitivityrange 505 to be produced when the luminance ratio becomes close to one,irrespective of the distance.

According to embodiments of the present disclosure to be describedbelow, it is possible to prevent the results of measurement from losingstability in those low-sensitivity ranges.

Embodiment 1

A first embodiment of a motion sensor device according to the presentdisclosure will be described. A motion sensor device according to thisembodiment includes three light sources.

FIG. 9A schematically illustrates an exemplary arrangement of lightsources in a motion sensor device including three light sources (whichwill be hereinafter referred to as first, second and third light sources102, 103 and 104, respectively). FIG. 9B is a timing chart showing thetimings for this motion sensor device to control the light sources andthe image sensor. First of all, it will be described with reference toFIGS. 9A and 9B why such a problem that the results of measurement wouldlose stability in the low-sensitivity ranges can be overcome by adoptingthe configuration of this embodiment.

The periods 802, 803 and 804 shown in FIG. 9B correspond to the periodsin which the LED light sources 102, 103 and 104 are respectively turnedON. The first, second and third exposure periods 805, 806 and 807correspond to the respective periods in which first, second and thirdframes are captured by the image sensor 101. Although the LED lightsources 102, 103 and 104 are supposed to be turned ON in this order inthe timing chart shown in FIG. 9B, the LED light sources may also beturned ON in any arbitrary order. Nevertheless, if the period in whichthe LED light source 102 is turned ON is defined to be the middle one ofthose periods in which the three light sources are respectively turnedON, decrease in measuring accuracy due to the object's motion can beminimized effectively.

An ordinary image sensor captures a single frame per exposure process,has image data thus obtained retrieved by an external device, and thencaptures the next frame. That is to say, an image data reading operationis performed on a frame-by-frame basis. With such an image sensor, inthe interval after an exposure process for the n^(th) frame (where n isan integer) has been finished and before an exposure process for the(n+1)^(th) frame is started, it will take some time to get the operationof transferring every electric charge obtained by capturing the n^(th)frame and outputting it to an external device done.

On the other hand, according to this embodiment, as soon as the firstexposure period 805 ends, the second exposure period 806 begins as shownin FIG. 9B. The electric charges of respective pixels which have beengenerated by capturing the first frame in the first exposure period 805are transferred to, and stored in, a storage section before the secondexposure period 806 begins. Likewise, as soon as the second exposureperiod 806 ends, the third exposure period 807 begins. The electriccharges of respective pixels which have been generated by capturing thesecond frame in the second exposure period 806 are transferred to, andstored in, another storage section before the third exposure period 807begins. After that, a signal representing the electric charges stored inthose storage sections and the electric charges generated in the thirdexposure period 807 is read and output to an external device in theperiod Tt.

According to this embodiment, if the length of the first to thirdexposure periods is Te, the data of three image frames is retrieved at arate to be determined by (1/Tf) which is the inverse number of Tf thatis as long as 3×Te+Tt.

The period of time Tt varies depending on the number of pixels but maybe set to be approximately 20 milliseconds with the data transfer ratetaken into account. On the other hand, the period of time Te may be setto be as short as 1 millisecond or less, e.g., 25 microseconds. If threeframes are sequentially captured within a short period of time, even thedistance to an object that is moving at high speeds (such as a person'sfingertip) can also be measured. For example, if 3×Te is 75microseconds, even an object that is moving at a speed of 1 meter persecond will move only 0.075 millimeters while the first to third framesare captured. On the other hand, if those frames are captured at anormal frame rate (of 60 frames per second, for example), then theobject will move as much as 50 millimeters in that period. Even if theobject is shot at as high speeds as 1000 frames per second, the objectwill still move 3 millimeters in that period. Since the period of timeafter the first frame has started and until the third frame ends can beshortened to 3 milliseconds or less according to this embodiment, such adevice can be used as a motion sensor device in various kinds ofapplications.

According to the configuration of this embodiment, the distance to theobject can be calculated based on two out of the three images that havebeen captured as the first to third frames. There are three differentcombinations of two images that can be chosen from given three images.And the positions of the low-sensitivity range are different from eachother in those three combinations. By using those two or three differentpairs of images, the low-sensitivity range can be removed. According tothis embodiment, the low-sensitivity range is supposed to be removedmainly by using two images captured as the first and second frames andtwo images captured as the second and third frames.

FIG. 10 schematically illustrates a low-sensitivity range to be producedin a situation where distance information has been collected from twoimages that have been captured as first and second frames and alow-sensitivity range to be produced in a situation where distanceinformation has been collected from two images that have been capturedas second and third frames. If the distance to the object is calculatedbased on two images that have been captured as the first and secondframes, the low-sensitivity range 605 is produced around an a point thatis located at equal distances from the light sources 102 and 103. On theother hand, if the distance to the object is calculated based on twoimages that have been captured as the second and third frames, thelow-sensitivity range 606 is produced around an a point that is locatedat equal distances from the light sources 103 and 104. Since the lightsource 103 is arranged according to this embodiment in the vicinity ofthe image sensor 101, no low-sensitivity range such as the one 504 shownin FIG. 8 will be produced at a close range.

The luminance of a frame which has been captured within a sufficientlyshort time under intense light emitted is substantially proportional tothe intensity of the reflected light. The luminances of the object thathas been shot as the first, second and third frames will be hereinafterreferred to as first, second and third luminances, respectively. Therespective luminances are determined by the angles and distances thatare defined by the relative positions of the object to the respectivelight sources. As described above, the distance to the object can beestimated based on the ratio of these luminances.

If the first and second luminances are compared to each other, theseluminances will turn out to be substantially equal to each other (i.e.,their luminance ratio will be close to one) in the low-sensitivity range605, and therefore, the distance measuring accuracy will decline there.If the second luminance is greater than the first luminance, probablythe object would be located on the right-hand side of thelow-sensitivity range 605 (i.e., closer to the light source 103).Conversely, if the first luminance is greater than the second luminance,probably the object would be located on the left-hand side of thelow-sensitivity range 605 (i.e., closer to the light source 102).

On the other hand, if the second and third luminances are compared toeach other, these luminances will turn out to be substantially equal toeach other (i.e., their luminance ratio will be close to one) in thelow-sensitivity range 606, and therefore, the distance measuringaccuracy will decline there. If the second luminance is greater than thethird luminance, probably the object would be located on the left-handside of the low-sensitivity range 606 (i.e., closer to the light source103). Conversely, if the third luminance is greater than the secondluminance, probably the object would be located on the right-hand sideof the low-sensitivity range 606 (i.e., closer to the light source 104).

In view of these considerations, the motion sensor device of thisembodiment gets a first piece of estimated distance information based onthe ratio of the first and second luminances, gets a second piece ofestimated distance information based on the ratio of the second andthird luminances, and either chooses one from, or combines together,these two pieces of information, thereby generating information aboutthe distance to the object. For example, if the ratio of the first andsecond luminances falls within a preset range, the device obtainsinformation about the distance to the object based on only the secondpiece of estimated distance information. On the other hand, if the ratioof the second and third luminances falls within a preset range, thedevice obtains information about the distance to the object based ononly the first piece of estimated distance information. In thisdescription, the “preset range” may be a predetermined range close toone (e.g., from 0.8 to 1.2). Also, the situations where the “ratio” ofthe two luminances “falls within a preset range” are supposed to includea situation where not the “ratio” but the “difference” between the twoluminances falls within a preset range.

Alternatively, the first piece of estimated distance information may beused only when the second luminance is greater than the first luminance,and the second piece of estimated distance information may be used onlywhen the second luminance is greater than the third luminance. Thedecision whether or not one luminance is greater than the other can bemade by either seeing if the absolute value of their difference isgreater than a predetermined threshold value or seeing if the differencebetween their ratio and one is greater than a predetermined thresholdvalue. By performing such a control, even if the object is located inthe low-sensitivity range 605 or 606, the distance to the object can bemeasured based on one of the two pieces of estimated distanceinformation. Optionally, although not adopted in this embodiment,estimated distance information may also be obtained based on a result ofcomparison between the first and third luminances.

As for a range where both of the first and second pieces of estimateddistance information have high sensitivity, the synthesis may be made bycalculating their average with some weight added to them. By performingsuch processing, the distance can be measured with high sensitivity andwith the low-sensitivity ranges removed.

As for the LED light source 103 to be arranged near the image sensor101, either its luminous intensity (radiation intensity) or luminousflux (radiant flux) value may be decreased or its radiation angle may beset to be a narrower one. Consequently, an inexpensive low-output LEDlight source can be used as the LED light source 104. And by adoptingsuch a light source, an increase in the cost of parts and powerdissipation can be checked. That is to say, according to thisembodiment, just by adding a single LED light source of a relatively lowprice, a motion sensor device with less low-sensitivity range isrealized.

In addition, according to this embodiment, by using a rather expensiveimage sensor which can capture three frames sequentially, either thedistance to an object that is moving at high speeds or thethree-dimensional motion of such an object can be detected. If themotion velocity of the object of measurement is expected to besufficiently low, an ordinary one-frame-exposure image sensor may beused.

Next, the configuration and operation of a motion sensor deviceaccording to this embodiment will be described in further detail withreference to FIGS. 11, 12 and 13.

FIG. 11 schematically illustrates a configuration for a motion sensordevice according to this embodiment. This device includes an imagesensor 101, a lens system 110 which produces a subject image on theimaging surface of the image sensor 101, three LED light sources 102,103 and 104, and a controller 1000 which is configured to control theimage sensor 101 and the LED light sources 102, 103 and 104. The imagesensor 101 and LED light sources 102, 103 and 104 are mounted on asubstrate 100. Part or all of the controller 1000 may be mounted on thesubstrate 100 or on another substrate. Alternatively, the function ofthe controller 1000 may be partially performed by a device which isarranged at a distant location.

The LED light source 103 is arranged closer to the image sensor 101 thanany other LED light source. Suppose the directions in which the LEDlight sources 102 to 104 are located with respect to the image sensor101 are first, second and third directions, respectively, and thedistances to the LED light sources 102 to 104 from the image sensor 101are first, second and third distances, respectively. The second andthird directions are opposite from the first direction. And the seconddistance is shorter than any of the first and third distances. In thisembodiment, the second and third distances are set to be equal to eachother.

The LED light sources 102, 103 and 104 are all configured to emit lightfalling within the same wavelength range. The image sensor 101 isconfigured to detect at least light falling within a particularwavelength range in the light emitted from the LED light sources 102,103 and 104. As the LED light sources 102, 103 and 104, ones that emitinvisible light such as a near-infrared ray can be used effectively fromthe standpoint of practice use. However, in applications in which thereis no problem even if visible light is emitted (e.g., in industrialapplications), the LED light sources 102, 103 and 104 may also beimplemented to emit visible light. The light sources do not have to beLED light sources but may also be point light sources or any other kindof light sources with a three-dimensionally biased intensitydistribution. In the following description, the LED light sources 102,103 and 104 are supposed to be configured to emit light with a firstwavelength (of 800 nm, for example) which is a near-infrared ray, andthe image sensor 101 is supposed to be configured to detect the lightwith the first wavelength.

In this description, the “near-infrared range” refers herein to awavelength range of about 700 nm to about 2.5 μm. Also, the“near-infrared ray” refers herein to light (electromagnetic wave), ofwhich the wavelength falls within the near-infrared range. Furthermore,“to emit light with the first wavelength” herein means emitting lightfalling within a broad wavelength range including the first wavelength.In this embodiment, the light sources 102, 103 and 104 do not have toemit light falling within exactly the same wavelength range, but theirwavelength ranges may be slightly different from each other as long asthe distance can be measured based on the luminance ratio.

According to this embodiment, by using the light sources 102 and 103,the distance to an object which is located just on the left-hand side ofthe image sensor 101 or anywhere on the right-hand side of the imagesensor 101 can be measured with good stability. Meanwhile, by using thelight sources 103 and 104, the distance to an object which is locatedjust on the right-hand side of the light source 103 or anywhere on theleft-hand side of the light source 103 can be measured with goodstability. In this manner, by using the luminance ratio obtained by thelight sources 102 and 103 and the luminance ratio obtained by the lightsources 103 and 104, the distance can be detected with much morestability with the low-sensitivity range eliminated.

The image sensor 101 includes a storage section which temporarily storeselectric charges on a pixel-by-pixel basis. Thus, even before image dataobtained by capturing an n^(th) frame is retrieved, an (n+1)^(th) framecan be captured. If an increased number of storage sections are providedinside the image sensor 101, the exposure process can be carried out onthree or more frames continuously. The image sensor 101 may be a specialkind of sensor which can carry out the exposure process on even-numberedlines and on odd-numbered lines separately from each other. Although theimage sensor of this embodiment is supposed to be a CMOS image sensor ora CCD image sensor, this is only an example and any other kind of imagesensor may also be used.

The controller 1000 is configured to make the image sensor 101 capture afirst frame with light emitted from the first light source 102 at afirst time, make the image sensor 101 capture a second frame with lightemitted from the second light source 103 at a second time, and make theimage sensor 101 capture a third frame with light emitted from the thirdlight source 104 at a third time. And the controller 1000 is configuredto obtain information about an estimated distance to the object 104based on multiple images generated by capturing the first to thirdframes.

FIG. 12 is a block diagram illustrating an exemplary configuration for amotion sensor device according to this embodiment.

The image capture device 1101 is a single-lens image capture device andincludes the image sensor 101 and lens system 110 shown in FIG. 11. Thelens system 110 may be a set of lenses which are arranged on the sameoptical axis. The light source unit 1102 includes the light sources 102,103 and 104 shown in FIG. 11.

The controller 1000 of this embodiment includes a CPU 1103 and asemiconductor integrated circuit 104, which includes a distancecalculating block 1105 and an image filter block 1106. The distancecalculating block 1105 includes an extreme value searching section 1107,a timing controller 1108, a coordinate memory 1109, and a distancecalculating section 1110.

Each of the light sources 102, 103 and 104 of this embodiment is an LEDlight source, and satisfies the relation between the angle of radiationand the relative radiation intensity shown in FIG. 2. It should be notedthat to carry out the present disclosure, the relative radiationintensity of these light sources 102, 103 and 104 does not have tochange with the angle of radiation. Actually, however, a lot of lightsources have a relative radiation intensity which changes with the angleof radiation. That is why this point is taken into considerationaccording to this embodiment.

In this embodiment, the timing controller 1108 shown in FIG. 12 sends asignal instructing that the LED light source 102 be turned ON to thelight source unit 1102. Meanwhile, the timing controller 1108 sends asignal instructing that an exposure process be carried out by the imagesensor 101 to the image capture device 1101. In this manner, an image iscaptured in the first frame with the light source 102 turned ON and sentto the semiconductor integrated circuit 1104.

Next, the timing controller 1108 sends a signal instructing that thelight source 103 be turned ON to the light source unit 1102. Meanwhile,the timing controller 1108 sends a signal instructing that an exposureprocess be carried out by the image sensor 101 to the image capturedevice 1101. In this manner, an image is captured in the second framewith the light source 103 turned ON and sent to the semiconductorintegrated circuit 1104.

Subsequently, the timing controller 1108 sends a signal instructing thatthe light source 104 be turned ON to the light source unit 1102.Meanwhile, the timing controller 1108 sends a signal instructing that anexposure process be carried out by the image sensor 101 to the imagecapture device 1101. In this manner, an image is captured in the thirdframe with the light source 104 turned ON and sent to the semiconductorintegrated circuit 1104.

In the semiconductor integrated circuit 1104, the image frame outputfrom the image capture device 1101 is processed by an image filter block1106. Although the image filter block 1106 is not an indispensableelement, pre-processing such as noise reduction filtering is performedaccording to this embodiment by the image filter block 1106 when imageprocessing is carried out.

The image processed by the image filter block 1106 is sent to thedistance calculating block 1105, in which the image is processed by theextreme value searching section 1107. An example of the data processedby the extreme value searching section 1107 is as shown in FIG. 5, inwhich shown is the luminance of the captured image on a predeterminedline. Although luminance distributions on the same line are shown inFIG. 5 with respect to two image frames, luminance distributions on thesame line are obtained according to this embodiment with respect tothree image frames. In other words, if two image frames are chosen fromthree image frames, two graphs, each of which is as shown in FIG. 5, canbe obtained. For example, a graph such as the one shown in FIG. 5 can bedrawn up based on first and second image frames. In the same way, agraph such as the one shown in FIG. 5 can also be drawn up based onsecond and third image frames.

The extreme value searching section 1107 searches first the range whereobjects to detect are present. There are a lot of searching methods. Forexample, it is easy to search for luminance extreme values 303 and 304based on the luminances 301 and 302 shown in FIG. 5. Alternatively, ifan extreme value which is stabilized sufficiently with respect to motionneeds to be obtained, it is possible to adopt a method in which a rangewhere the luminance is equal to or greater than a certain value isdetected and its center value is regarded as an extreme value.

Next, the extreme values 303 and 304 are regarded as having beenobtained from the same object and paired with each other. In this case,two extreme values with close coordinates may be simply paired with eachother. Or a range in which the luminance is equal to or higher than acertain value may be located in advance based on the sum of theluminances 301 and 302 and may be searched for extreme values.

Check the difference between the luminances 301 and 302 shown in FIG. 5,and it can be seen that there is a range with a luminance level eventhough there is no difference there. However, as such a range existsprobably because an out-of-system light source is present outside ofthis device, that range could be regarded as disturbance factor andremoved. If cost permits, an image may be captured with every lightsource of this system turned OFF and that range may be removed. Even so,the same effect can also be achieved.

The extreme value searching section 1107 outputs the coordinates andextreme values of the object detected. The coordinates may be those ofthe center or barycenter of the range 306 or those of the middle pointbetween the extreme values 303 and 304. Meanwhile, the extreme values303 and 304 may be used as they are as the luminances according to onemethod, or an integral value of the range may be obtained according toanother method.

In this description, one-dimensional data on a particular line has beendescribed for the sake of simplicity. However, the one-dimensional linemay be an axis other than the horizontal line for capturing an image.Alternatively, the coordinates and luminances of a range with a highrelative luminance level may also be searched for two-dimensionally.

The coordinates and extreme values of the object that have been outputfrom the extreme value searching section 1107 are stored in thecoordinate memory 1109 and then sent to the distance calculating section1110.

The distance calculating section 1110 calculates the distance based onthe ratio of the luminances that have been obtained from the first andsecond image frames. First of all, based on the coordinates of theobject, the distance calculating section 1110 determines in what azimuththe object is located with respect to the image sensor 101. This azimuthcan be determined uniquely with the property of an optical system suchas a lens taken into account.

Next, when it is known at what distance the object is located in thatazimuth, the three-dimensional position of the object can be estimated.

The radiation characteristic of an LED light source that changes withits position such as the one shown in FIG. 6 mentioned above has beenobtained in each azimuth. The data shown in FIG. 6 is based on the factthat the intensity of light decreases inversely proportionally to thesquare of the distance between an LED light source and the object. Also,in order to increase the accuracy, the angle defined by the object ateach distance with respect to the LED light source is corrected inaccordance with the radiation characteristic shown in FIG. 2. If thedata shown in FIG. 6 is available, the distance to the object can becalculated based on the luminance ratio.

The data shown in FIG. 6 may be calculated by the distance calculatingsection 1110 through trigonometric function calculation. Alternatively,the data shown in FIG. 6 may also be calculated by storing, as a table,a graph which has been obtained in advance by calculation or measurementand complementing the graph as needed.

The results obtained by the distance calculating block 1105 are suppliedto the CPU 103 and used as 3D motion information there.

According to the configuration described above, processing can beadvanced on the image data on a line-by-line basis. As a result, amotion sensor device which can detect the object in only one path withlittle latency is realizable.

The coordinates of the extreme values 303 and 304 do not always have toagree with each other. But as long as the material of the object isroughly uniform within the object area, these extreme values 303 and 304can be used as the luminance ratio for calculating the distance.Optionally, the unit of measurement may be defined to be only the unitof an object with a certain width with attention paid to this property.According to this embodiment, the extreme values are searched for first,and then the distance is calculated based on the extreme valuesobtained. In this manner, the computations can get done more speedilywith its complexity reduced significantly.

For example, in measuring the conditions of respective limbs of a humanbody, the extreme values of luminances of the respective regions thathave been shot are obtained on an arm, leg or neck basis by reference tothe data on a certain line. That is why compared to a method ofcalculating some distance at each pixel, the number of times ofcomputations to get done can be reduced significantly.

Up to this point, the processing described above can get done with onlythe CPU and a software program. The processing flow of a softwareprogram to be executed in that case is shown in FIG. 13. This processingincludes an extreme value searching step 1201, a threshold valuedetermining step 1202, a luminance ratio calculating step 1203 and adistance converting step 1204.

In the extreme value searching step 1201, the image data is searched fora range with a relatively high luminance value (i.e., a range includingan extreme value). Next, in the threshold value determining step 1202,the decision is made whether or not the given object is the object to betracked in the extreme value searching step 1201. If the luminance orsize of the range is equal to or smaller than a certain value, then thedecision is made that “there is no object” and the data is regarded asnoise and discarded. This threshold value determining step 1202 is notan indispensable step but is generally an important step to increase therobustness. On the other hand, if the decision made in the thresholdvalue determining step 1202 is that “there is an object”, thenassociated extreme values are paired with each other to calculate theluminance ratio. Subsequently, in the distance converting step 1204, theextreme values are converted into a distance based on the luminanceratio and the image capturing position.

Alternatively, this function can also be performed by storing a programdefining the procedure described above on a magnetic recording medium ora semiconductor storage medium, for example, and getting the programdone by the CPU.

According to this embodiment, by scanning the image only once in theextreme value searching step 1201, the luminance value and coordinatesto be the object of calculation can be picked up. That is why byadopting this procedure, the computations can get done speedily.

The motion sensor device of this embodiment can be used in variousapplications. For example, by applying this motion sensor device to acamcorder, movie autofocusing can be controlled quickly. In addition, byrecognizing respective fingers of a human being from a short distance orhis or her body or limbs from a long distance, this device can also beused as a gesture recognizing motion sensor device.

In the embodiment described above, the light sources 102, 103 and 104 donot have to have the same height and same size but may have differentheights or sizes. In addition, each of these light sources 102, 103 and104 does not have to be a single LED chip, either. Alternatively, an LEDarray in which a number of LED chips are arranged may be used as each ofthose light sources. Furthermore, although not shown, optical memberssuch as a lens and a filter may be arranged in each of those lightsources 102, 103 and 104. The same can be said about the light sourcesof any of the other embodiments.

The light sources 102, 103 and 104 and the image sensor 101 do not haveto be arranged in line. FIGS. 14A and 14B are top views illustratingalternative arrangements for the light sources 102, 103 and 104 and theimage sensor 101. Some of the light sources 102, 103 and 104 (e.g., thelight source 103 in this example) may be arranged off the line as shownin FIG. 14A. Alternatively, one light source and the image sensor 101may be arranged in line but the other light sources may be arranged offthat line as shown in FIG. 14B.

In the example illustrated in FIG. 14A, two light sources 102 and 104are arranged at the same distance but in mutually opposite directionswith respect to the image sensor 101. However, those light sources 102and 104 may also be arranged at different distances. On the other hand,in the example illustrated in FIG. 14B, the three light sources 102, 103and 104 are arranged in mutually different directions with respect tothe image sensor 101. When the light sources 102, 103 and 104 arearranged in different directions in this manner, the distances from theimage sensor 101 to the respective light sources 102, 103 and 104 do nothave to be different from, but may be equal to, each other.

As can be seen from the foregoing description, the controller 1000 ofthis embodiment gets a first piece of estimated distance informationbased on the ratio of an object's luminance obtained from a first imageto the object's luminance obtained from a second image, and gets asecond piece of estimated distance information based on the ratio of theobject's luminance obtained from a second image to the object'sluminance obtained from a third image. And the controller 1000 getsinformation about the distance to the object by either choosing onefrom, or synthesizing together, the first and second pieces of estimateddistance information. However, the controller 1000 may get informationabout the distance to the object by performing a different operationfrom such an operation. For example, the controller 1000 can also obtainthe distance to the object by either choosing one from, or synthesizingtogether, pieces of luminance information of multiple images yet to beconverted into the distance.

Specifically, information about the distance to the object can be gottenbased on the ratio of a fourth luminance which is obtained by mixingtogether the first and third luminances at a predetermined ratio to thesecond luminance. For example, suppose the first luminance is P1, thethird luminance is P3, the mixing ratio is a (where 0<a<1) and thefourth luminance is P1×a+P3×(1−a). In that case, based on the ratio ofthe fourth luminance to the second luminance P2 (i.e.,(P1×a+P3×(1−a))/P2), information about the distance to the object can beobtained by reference to the relation between the known luminance ratioand the distance. According to this method, the luminance mixing ratio amay be determined with respect to each pixel location in the imagesensor 101.

Still alternatively, either the first luminance or the third luminancemay be chosen and regarded as the fourth luminance, and the distance maybe calculated in the same way as described above. This corresponds to asituation where the mixing ratio a described above is zero or one.Either the first luminance or the third luminance is chosen bydetermining whether or not the ratio or difference between the twoluminances falls within a predetermined range.

Embodiment 2

A second embodiment of a motion sensor device according to the presentdisclosure will be described. The motion sensor device of thisembodiment includes four light sources (which will be hereinafterreferred to as first, second, third and fourth light sources 702, 703,704 and 705, respectively). The device of this embodiment also includesthe lens system 110 and controller 1000 with the same configurations asthe ones already described with reference to FIG. 11. Thus, descriptionthereof will be omitted herein to avoid redundancies.

FIG. 15A illustrates an arrangement of light sources in the motionsensor device of this embodiment. FIG. 15B is a timing chart showing thetimings for this motion sensor device to control the light sources andthe image sensor.

The motion sensor device of this embodiment includes an image sensor 701and four LED light sources 702, 703, 704 and 705 which are mounted on asubstrate 700. The LED light sources 702 and 703 are configured to emitlight with a first wavelength. The LED light sources 704 and 705 areconfigured to emit light with a second wavelength. The image sensor 701is configured to be able to detect at least the light with the firstwavelength and the light with the second wavelength.

The first and second wavelengths may be arbitrary wavelengths but aresupposed to be 780 nm and 850 nm, respectively, in the followingdescription. Both of these are wavelengths falling within thenear-infrared range. However, the first and second wavelengths do nothave to be wavelengths falling within the near-infrared range but mayalso be visible light wavelengths, for example. From the standpoint ofpractical use, the first and second wavelengths are suitably set to bewavelengths falling within the near-infrared range. However, inapplications in which there is no problem even if visible light isemitted (e.g., in industrial applications), the first and secondwavelengths may be visible light wavelengths, too.

The directions in which the light sources 702 to 705 are located withrespect to the image sensor 701 are supposed to be first through fourthdirections, respectively, and the distances to the light sources 702 to705 are supposed to be first through fourth directions, respectively.The second and fourth directions are opposite from the first direction.The third direction is the same as the first direction. Both of thesecond and third distances are shorter than the first and fourthdistances. In this embodiment, the second and third distances are equalto each other, so are the first and fourth distances. However, sucharrangements do not always have to be adopted.

The image sensor 701 is a special sensor which can capture two frames ina row by performing exposure processes twice continuously. The firstembodiment described above needs a special sensor which can performexposure processes three times in a row. Such a sensor should work finein principle but it would take a lot of cost to make such a sensoractually. According to this embodiment, the distance can be measuredwith a configuration of a lower cost. In addition, according to thisembodiment, not just can the distance be measured but also can thematerial of the object be determined as will be described later.

The periods 812, 813, 814 and 815 shown in FIG. 15B correspond to theperiods in which the LED light sources 702, 703, 704 and 705 arerespectively turned ON. The first and second exposure periods 816 and817 correspond to the respective periods in which first and secondframes are captured by the image sensor 101. According to the timingchart shown in FIG. 15B, the LED light sources 702 and 705 are turned ONsimultaneously first, and then the LED light sources 703 and 704 areturned ON simultaneously. However, this is only an example. Instead, theemission may be controlled so that one of the LED light sources 702 and703 that emit light with the first wavelength and one of the LED lightsources 704 and 705 that emit light with the second wavelength areturned ON simultaneously.

If images are captured in accordance with the timing chart shown in FIG.15B, the controller gets a first piece of estimated distance informationbased on the ratio of the luminance of an object's first wavelengthcomponent obtained from a first image produced by capturing the firstframe to the luminance of the object's first wavelength componentobtained from a second image produced by capturing the second frame.Also, the controller gets a second piece of estimated distanceinformation based on the ratio of the luminance of the object's secondwavelength component obtained from the first image to the luminance ofthe object's second wavelength component obtained from the second image.And the controller generates and outputs information about the distanceto the object by either choosing one from, or synthesizing together, thefirst and second pieces of estimated distance information.

FIG. 16 schematically illustrates low-sensitivity ranges associated withthe first and second pieces of estimated distance information. If onlythe first piece of estimated distance information is used, alow-sensitivity range 711 is produced around a region which is locatedat an equal distance from the light sources 702 and 703. On the otherhand, if only the second piece of estimated distance information isused, a low-sensitivity range 712 is produced around a region which islocated at an equal distance from the light sources 704 and 705. Sincethe light sources 703 and 704 are arranged in this embodiment in thevicinity of the image sensor 101, the low-sensitivity range 504 at aclose range shown in FIG. 8 is not produced.

The motion sensor device of this embodiment either chooses one from, orsynthesizes together, the first and second pieces of estimated distanceinformation, and therefore, can measure the distance to even an objectlocated in the low-sensitivity range 711, 712. Specifically, if theratio of the luminance of the object's first wavelength componentobtained from the first image to the luminance of the object's firstwavelength component obtained from the second image falls within apreset range (e.g., from 0.8 to 1.2), the controller gets informationabout the distance to the object based on only the second piece ofestimated distance information. On the other hand, if the ratio of theluminance of the object's second wavelength component obtained from thefirst image to the luminance of the object's second wavelength componentobtained from the second image falls within a preset range, thecontroller gets information about the distance to the object based ononly the first piece of estimated distance information. Alternatively,the controller may use the first piece of estimated distance informationonly when the luminance of the object's first wavelength componentobtained from the second image is greater than the luminance of theobject's first wavelength component obtained from the first image. Andthe controller may use the second piece of estimated distanceinformation only when the luminance of the object's second wavelengthcomponent obtained from the second image is greater than the luminanceof the object's second wavelength component obtained from the firstimage. By performing such processing, the distance to the object can bemeasured accurately irrespective of that object's position.

The image sensor 701 may be a sensor which has multiple kinds of pixelswith different spectral sensitivity characteristics just like a knowncolor image sensor. There are various kinds of color image sensors,examples of which include a sensor with a Bayer arrangement in which R,G, G and B pixels are arranged as a unit consisting of four pixels, asensor which can get signal electric charges of multiple colorcomponents in a single pixel by taking advantage of the fact that thetransmission characteristic varies according to the wavelength, and asensor in which incoming light is split through a prism on acolor-by-color basis into multiple light beams to be received by aplurality of image capture devices on a color component basis. No matterwhich of these color image sensors is used, the light can be received tohave its spectral sensitivity characteristic varied according to thecolor component, and therefore, the color image sensor may be used asthe image sensor 701.

FIG. 17 is a graph showing exemplary spectral sensitivitycharacteristics of the image sensor 701 according to this embodiment ona color component basis. In the graph shown in FIG. 17, the abscissarepresents the wavelength and the ordinate represents the relativesensitivity value. This image sensor 701 is supposed to have an R pixelwhich detects mainly a color red component of the light, a G pixel whichdetects mainly a color green component, and a B pixel which detectsmainly a color blue component. As shown in FIG. 17, each of the R, G andB pixels' spectral sensitivity characteristics 901, 902 and 903 is alsosensitive to a wavelength falling within the near-infrared range. In thefollowing example, it will be described how to obtain the respectiveluminance values of the first and second wavelength components by usingthe R and B pixels' sensitivities 911 and 913 to light with the firstwavelength and the R and B pixels' sensitivities 912 and 914 to lightwith the second wavelength.

Suppose the R pixel's sensitivity 911 to the light with the firstwavelength is “a”, the R pixel's sensitivity 912 to the secondwavelength is “b”, the B pixel's sensitivity 913 to the first wavelengthis “c”, and the B pixel's sensitivity 914 to the second wavelength is“d”. Also, suppose a luminance corresponding to the intensity of lightwith the first wavelength being incident on a certain pixel in one frameis “α” and a luminance corresponding to the intensity of light with thesecond wavelength is “β”. In that case, the red and blue components rand b of the luminance of a certain pixel in one frame are representedby the following Equation (1):

$\begin{matrix}{\begin{pmatrix}r \\b\end{pmatrix} = {\begin{pmatrix}a & b \\c & d\end{pmatrix}\begin{pmatrix}\alpha \\\beta\end{pmatrix}}} & (1)\end{matrix}$

Thus, by obtaining the inverse matrix of the matrix of Equation (1), theluminances α and β of the first and second wavelength components of thatpixel can be calculated by the following Equation (2):

$\begin{matrix}{\begin{pmatrix}\alpha \\\beta\end{pmatrix} = {\begin{pmatrix}a & b \\c & d\end{pmatrix}^{- 1}\begin{pmatrix}r \\b\end{pmatrix}}} & (2)\end{matrix}$

By performing these computations on each of the first and second frames,the luminances of the first and second wavelength components in eachframe can be obtained. Although the R and B pixels' spectral sensitivitycharacteristics are supposed to be used in this example, computationsmay be performed in the same way with one of these spectral sensitivitycharacteristics replaced with the G pixel's. Also, even when using animage sensor which can obtain signal electric charges of multiple colorcomponents in a single pixel or an image sensor which splits theincoming light through a prism on a color-by-color basis and which getsthe resultant light beams received by multiple image capture devices ona color component basis by taking advantage of the fact that thetransmission characteristic varies according to the wavelength,computations may be performed in the same way based on thecolor-by-color spectral sensitivity characteristics of the incidentlight.

As described above, the controller of this embodiment uses therespective luminances of the first wavelength components extracted fromthe first and second frames and calculates, based on the ratio of thoseluminances, the distance to the object as a first piece of estimateddistance information. In this case, in the low-sensitivity range 711 inwhich the luminance ratio is close to one, the distance cannot bemeasured accurately. Meanwhile, the LED light source 703 is located soclose to the image sensor 701 that there is almost no range where therelation between the light intensities inverts, and the distance can bemeasured accurately in a broad range, on the right-hand side of thelow-sensitivity range 711.

The controller also uses the respective light intensity components withthe second wavelength extracted from the first and second frames andcalculates, based on the ratio of those light intensities, the distanceto the object as a second piece of estimated distance information. Inthis case, in the low-sensitivity range 712 in which the luminance ratiois close to one, the distance cannot be measured accurately. Meanwhile,the LED light source 704 is located so close to the image sensor 701that there is almost no range where the relation between the lightintensities inverts, and the distance can be measured accurately in abroad range, on the left-hand side of the low-sensitivity range 712.

Thus, it can be seen that the first piece of estimated distanceinformation obtained based on the luminance ratio of the firstwavelength components and the second piece of estimated distanceinformation obtained based on the luminance ratio of the secondwavelength components complement the low-sensitivity range with eachother. That is why the controller complements a range in which thesensitivity is low according to one of these two pieces of estimateddistance information with the other piece of information, andsynthesizes those pieces of information together (e.g., calculates theiraverage with a weight added) as for a range where the sensitivity ishigh according to both of these pieces of information. By performingsuch processing, all of those ranges can be integrated togetherseamlessly, and the distance can be measured with high sensitivity.

According to this embodiment, the object is shot at two wavelengths(i.e., at the first and second wavelengths), and therefore, not only canthe distance information be estimated but also can the object's materialbe determined as well. Next, it will be described how to determine thematerial in principle according to this embodiment.

Generally speaking, anything that exists in Nature has its absorptancevaried with the wavelength, and the reflectance of the light that hasnot been absorbed also varies with the wavelength. That is why even iftwo light beams have been emitted at the same intensity but if theirwavelengths are different, the reflected light beams may have differentintensities. In the visible light range, their difference is recognizedas a color difference. Even in the non-visible light range, the materialof the object can be estimated by sensing a difference in intensitybetween reflected light beams.

As disclosed in Non-Patent Document No. 1, it is known that the humanskin exhibits low reflectance to a near-infrared ray, of which thewavelength is in the vicinity of 970 nm. That is why just by comparingthe levels of the reflected light intensity of that near-infrared ray inthe vicinity of 970 nm and the reflected light intensity of anear-infrared ray in the vicinity of 870 nm, at which the reflectance isrelatively high, the estimation can be made whether the object would bea human skin or not. This estimation can be made not just at thosewavelengths of around 970 nm and around 870 nm but also at any otherwavelength as well.

Thus, the controller of this embodiment determines the material of theobject based on either the difference or ratio between the respectiveluminances of the object's first and second wavelength components in atleast one of the first and second images. For example, the controllermay determine whether or not the difference between the respectiveluminances of the object's first and second wavelength components agreeswith the difference in human skin reflectance to light beams with thefirst and second wavelengths. By performing such processing, thedecision can be made whether or not the object is a human hand. Althougha human hand is supposed to be detected as target in this example,decision can be made in the same way with respect to a pointer or anyother target as well.

According to a more sophisticated method, an additional light source foruse to estimate the material (which will be hereinafter referred to as a“fifth light source”) may be further provided besides the ones for useto measure the distance. The material-estimating light source may beconfigured to emit light with a third wavelength (of 970 nm, forexample) which is different from the two wavelengths (i.e., the firstand second wavelengths) for measuring the distance. Optionally, thematerial-estimating light source may be a combination of multiple lightsources which emit light beams with mutually different wavelengths. Theobject's material can be determined based on the difference in reflectedlight intensity between at least two light beams with differentwavelengths. For example, supposing the two distance-measuringwavelengths are λ1 and λ2 and the material-estimating wavelength is λ3,the object's material can be estimated based on the ratio of therespective luminances at these three wavelengths. Alternatively, byusing three light sources which emit light beams at the wavelengths λ1,λ2 and λ3, the object's material can also be determined based on theratio of the luminances at these three wavelengths. More specifically,using a color image sensor which detects light beams falling within theRGB wavelength ranges and three light sources which emit light beamsfalling within the RGB wavelength ranges, the material can be determinedbased on the ratio of the respective luminances of these threecomponents. If multiple conditions have been set in advance inassociation with a plurality of materials with respect to the ratio ofthe luminances of these three components, the decision can be madeselectively with respect to those materials.

Although the material-estimating light source is supposed to be providedseparately from the distance-measuring light sources in the exampledescribed above, these two types of light sources may be integratedtogether. That is to say, a light source which emits light fallingwithin a broad wavelength range covering not only the first and secondwavelengths but also the third wavelength as well may be used to measurethe distance and to estimate the material. If at least one of the firstthrough fourth light sources 702 to 705 is such a light source, thematerial can be determined in the same way as in a situation where thefifth light source is added.

In one application, a 3D motion sensor may be used as a user interfaceto detect only a human finger or pointer, for example. When a pointer isbeing tracked, for example, information about a human finger may need tobe eliminated as wrong information. In that case, the motion sensor ofthis embodiment can determine, based on the difference in luminancebetween at least two wavelength components, whether the object is apointer or a finger. Also, if a pointer is made of a dedicated materialwith a characteristic spectral reflection property, such a pointer willcause much less erroneous detection.

As can be seen, according to this embodiment, based on the differencebetween the light intensities (or luminances) of at least two wavelengthcomponents obtained in the same frame, not just can the distance bemeasured but also can the object's material be estimated as well withoutnewly adding another resource. By using light with a differentwavelength as needed, multiple different kinds of materials can bedetermined as well.

Although the first and second wavelengths are both supposed to fallwithin the near-infrared range in the embodiment described above, thisis only an example. The wavelength settings described above are adoptedbecause it is often convenient to use non-visible light which isinvisible to human beings, considering the property of a motion sensor.However, the measurement itself may be made with visible light as well.For example, in an application in which a non-human object is the objectof measurement, the visible light wavelength range with high sensitivitymay also be used.

Also, a camera which uses a color image sensor ordinarily cuts lightfalling within the infrared range by adding an infrared cut filter toits optical system. According to this embodiment, on the other hand, theinfrared cut filter can be omitted, and therefore, a motion sensordevice is realized at a lower cost. In addition, according to thisembodiment, an ordinary color image sensor for shooting under visiblelight can also be used. Naturally, an image sensor including a colorfilter with a good performance which is specially designed for thisembodiment may be used, but the cost usually increases in that case.

Although LED light sources are used in the embodiment described above,the light sources do not have to be LED light sources but may also bepoint light sources or any other kind of light sources with athree-dimensionally varying intensity distribution. Since thewavelengths need to be defined, it is also effective to use laser lightsources. A laser light source emits parallel light, and therefore, itslight intensity does not vary three-dimensionally. However, a laserlight source may also be used if the parallel light is turned intoscattering light by combining the laser light source with a diffuser,for example.

Furthermore, although the second and third LED light sources 703 and 704are handled separately in the embodiment described above, these twolight sources may be integrated together. Since these light sources arearranged close to each other and may be made to emit light at the sametiming in one embodiment, a single light source unit which emits lightfalling within a broad wavelength range covering the first and secondwavelengths may be used as the second and third light sources.

As described above, the controller 1000 of this embodiment is configuredto generate a first piece of estimated distance information based on theratio of the luminance of an object's first wavelength componentobtained from a first image to the luminance of the object's firstwavelength component obtained from a second image, generate a secondpiece of estimated distance information based on the ratio of theluminance of the object's second wavelength component obtained from thefirst image to the luminance of the object's second wavelength componentobtained from the second image, and get information about the distanceto the object by either choosing one from, or synthesizing together, thefirst and second pieces of estimated distance information. However, thecontroller 1000 does not always have to operate in this manner but mayget information about the distance to the object by operating in anyother manner as well. For example, if the object's first wavelengthcomponents obtained from the first image and the second image producedby capturing a second frame are called first and second luminances,respectively, and if the object's second wavelength components obtainedfrom the first and second images are called third and fourth luminances,respectively, the controller 1000 may be configured to get informationabout the distance to the object based on the ratio of a fifth luminancewhich is obtained by either choosing one from, or synthesizing together,the first and second luminances to a sixth luminance which is obtainedby either choosing one from, or synthesizing together, the third andfourth luminances.

Specifically, the fifth luminance may be defined by mixing the first andsecond luminances at a predetermined mixing ratio, the sixth luminancemay be defined by mixing the third and fourth luminances at apredetermined mixing ratio, and information about the distance to theobject can be gotten based on the ratio of the fifth and sixthluminances. According to this method, the luminance mixing ratio may beadjusted on a pixel location basis in the image sensor 101.

Alternatively, one chosen from the first and second luminances may bedefined to be the fifth luminance, one chosen from the third and fourthluminances may be defined to be the sixth luminance, and the distancemay be calculated in the same way as described above. The choice of oneof the two luminances is made by determining whether or not the ratio ordifference between those two luminances falls within a predeterminedrange.

Other Embodiments

Although Embodiments 1 and 2 have been described, these are justexamples of the technique of the present disclosure. Thus, some otherexemplary embodiments will be described.

FIG. 18 illustrates a display 1001 with a motion sensor device accordingto the first embodiment. This display 1001 includes three LED lightsources 102, 103 and 104, and therefore, can sense motion with thelow-sensitivity range cut down. That is why a gesture input can be madetoward the center of the display 1001. In FIG. 18, illustratedschematically is a hand making the gesture input for your reference. Thehand illustrated in FIG. 18 is moving in the direction indicated by thearrow. The display shown in FIG. 18 can sense such a motion of the hand(i.e., a gesture input) with high sensitivity.

If the motion sensor device of this embodiment is applied to a display,for example, the device can be used as a user interface which allows theuser to change channels with a gesture input. This motion sensor deviceis also applicable to a dance game to recognize the motion of respectivelimbs of a human being.

Alternatively, a motion sensor device according to the second embodimentor any other embodiment may be built in the display shown in FIG. 18. Ascan be seen, the present disclosure may be implemented as an electronicdevice including a motion sensor device according to any of theembodiments described above and a display which changes the content tobe presented thereon in response to the object's motion that has beendetected by the motion sensor device.

A motion sensor device according to various embodiments of the presentdisclosure can reduce errors in measuring the distance and can operatemuch more quickly. A 3D motion sensor device according to the presentdisclosure can be used in applications in which detection needs to bedone in real time. In addition, according to one aspect, not only canthe distance be measured but also can the material of the object bedetermined as well. As a result, the present disclosure realizes amotion sensor device which can detect a human hand or a pointer witherroneous detection reduced significantly.

Optionally, some of the functions of a motion sensor device according tothe present disclosure may be performed by another device which isconnected to the former device through a wired or wireless network.

INDUSTRIAL APPLICABILITY

An embodiment of a motion sensor device according to the presentdisclosure has the ability to measure the three-dimensional position ofan object in real time, and therefore, can be used effectively as anon-contact gesture user interface for a display device and variousother kinds of electronic devices. In addition, this motion sensordevice may also be used as a car device to monitor the state ofperson(s) inside the car and persons outside of the car and to detectany obstacles. Furthermore, the motion sensor device can also be used inautofocusing for a camcorder.

REFERENCE SIGNS LIST

-   101, 701 image sensor-   102, 103, 104, 702, 703, 704, 705 LED light source-   301, 302 luminance-   303, 304 extreme value-   305 difference in coordinate between extreme values of luminances-   504, 505, 605, 606, 711, 712 low-sensitivity range-   802, 803, 804, 812, 813, 814, 815 light-emitting period-   805, 806, 807, 816, 817 exposure period-   901, 902, 903 pixel's spectral sensitivity characteristic-   1001 display device-   1101 image capture device-   1102 light source unit-   1103 CPU-   1104 semiconductor integrated circuit-   1105 distance calculating block-   1106 image filter block-   1107 extreme value searching section-   1109 coordinate memory-   1110 distance calculating section

1. A motion sensor device comprising: an image sensor; first, second andthird light sources; and a controller configured to control the imagesensor and the first to third light sources, wherein the controller isconfigured to: make the image sensor capture a first frame with lightemitted from the first light source at a first time; make the imagesensor capture a second frame with light emitted from the second lightsource at a second time; make the image sensor capture a third framewith light emitted from the third light source at a third time; generatea first piece of estimated distance information based on a ratio of aluminance of an object obtained from a first image produced by capturingthe first frame to a luminance of the object obtained from a secondimage produced by capturing the second frame; generate a second piece ofestimated distance information based on a ratio of the luminance of theobject obtained from the first image to a luminance of the objectobtained from a third image produced by capturing the third frame; andobtain information about a distance to the object by either choosing onefrom, or synthesizing together, the first and second pieces of estimateddistance information.
 2. The motion sensor device of claim 1, whereinthe first light source is arranged at a first distance from the imagesensor in a first direction, the second light source is arranged at asecond distance from the image sensor in a second direction, the thirdlight source is arranged at a third distance from the image sensor in athird direction, the second and third directions are opposite from thefirst direction, and the second distance is shorter than any of thefirst and third distances.
 3. The motion sensor device of claim 2,wherein the controller is configured to obtain information about thedistance to the object based on only the second piece of estimateddistance information when the ratio of the luminance of the objectobtained from the first image to the luminance of the object obtainedfrom the second image falls within a preset range, and obtaininformation about the distance to the object based on only the firstpiece of estimated distance information when the ratio of the luminanceof the object obtained from the second image to the luminance of theobject obtained from the third image falls within a preset range.
 4. Themotion sensor device of claim 3, wherein the controller is configured toobtain information about the distance to the object based on the firstpiece of estimated distance information only when the luminance of theobject obtained from the second image is greater than the luminance ofthe object obtained from the first image, and obtain information aboutthe distance to the object based on the second piece of estimateddistance information only when the luminance of the object obtained fromthe second image is greater than the luminance of the object obtainedfrom the third image.
 5. The motion sensor device of claim 1, whereinthe image sensor is configured to be able to detect light with a firstwavelength which is a near-infrared ray, and the first to third lightsources are configured to emit light with the first wavelength.
 6. Amotion sensor device comprising: an image sensor which is able to detectlight with a first wavelength and light with a second wavelength that isdifferent from the first wavelength; first and second light sourceswhich emit light with the first wavelength; third and fourth lightsources which emit light with the second wavelength; and a controllerconfigured to control the image sensor and the first to fourth lightsources, wherein the controller is configured to: make the image sensorcapture a first frame with light emitted from the first and fourth lightsources at a first time; make the image sensor capture a second framewith light emitted from the second and third light sources at a secondtime; generate a first piece of estimated distance information based ona ratio of a luminance of a first wavelength component of an objectobtained from a first image produced by capturing the first frame to aluminance of the first wavelength component of the object obtained froma second image produced by capturing the second frame; generate a secondpiece of estimated distance information based on a ratio of a luminanceof a second wavelength component of the object obtained from the firstimage to a luminance of the second wavelength component of the objectobtained from the second image; and obtain information about thedistance to the object by either choosing one from, or synthesizingtogether, the first and second pieces of estimated distance information.7. The motion sensor device of claim 6, wherein the first light sourceis arranged at a first distance from the image sensor in a firstdirection, the second light source is arranged at a second distance fromthe image sensor in a second direction, the third light source isarranged at a third distance from the image sensor in a third direction,the fourth light source is arranged at a fourth distance from the imagesensor in a fourth direction, the second and fourth directions areopposite from the first direction, the third direction is the same asthe first direction, and each of the second and third distances isshorter than any of the first and fourth distances.
 8. The motion sensordevice of claim 6, wherein the controller is configured to obtaininformation about the distance to the object based on only the secondpiece of estimated distance information when the ratio of the luminanceof the first wavelength component of the object obtained from the firstimage to the luminance of the first wavelength component of the objectobtained from the second image falls within a preset range, and obtaininformation about the distance to the object based on only the firstpiece of estimated distance information when the ratio of the luminanceof the second wavelength component of the object obtained from the firstimage to the luminance of the second wavelength component of the objectobtained from the second image falls within a preset range.
 9. Themotion sensor device of claim 6, wherein the second and third lightsources are implemented as a single integrated light source unit. 10.The motion sensor device of claim 6, wherein the first and secondwavelengths both fall within a near infrared range.
 11. The motionsensor device of claim 6, wherein the controller is configured todetermine a material of the object based on a difference in luminancebetween the first and second wavelength components of the object in atleast one of the first and second images.
 12. The motion sensor deviceof claim 6, further comprising a fifth light source which emits lightwith a third wavelength that is different from the first and secondwavelengths, wherein the image sensor is configured to be able to detectlight with the third wavelength, too, and the controller is configuredto make the image sensor capture an image with light also emitted fromthe fifth light source in at least one of the first and second times,and determine the material of the object based on either a difference orratio between the third wavelength component of the object and at leastone of the first and second wavelength components which are obtainedfrom at least one of the first and second images.
 13. The motion sensordevice of claim 12, wherein the fifth light source is integrated withone of the first to fourth light sources.
 14. A motion sensor devicecomprising: an image sensor; first, second and third light sources; anda controller configured to control the image sensor and the first tothird light sources, wherein the controller is configured to: make theimage sensor capture a first frame with light emitted from the firstlight source at a first time; make the image sensor capture a secondframe with light emitted from the second light source at a second time;and make the image sensor capture a third frame with light emitted fromthe third light source at a third time, and wherein luminances of anobject obtained from first, second and third images produced bycapturing the first, second and third frames are called first, secondand third luminances, respectively, the controller is configured toobtain information about the distance to the object based on a ratio ofa fourth luminance which is obtained by either synthesizing together, orchoosing one from, the first and third luminances to the secondluminance.
 15. A motion sensor device comprising: an image sensor whichis able to detect light with a first wavelength and light with a secondwavelength that is different from the first wavelength; first and secondlight sources which emit light with the first wavelength; third andfourth light sources which emit light with the second wavelength; and acontroller configured to control the image sensor and the first tofourth light sources, wherein the controller is configured to: make theimage sensor capture a first frame with light emitted from the first andfourth light sources at a first time; and make the image sensor capturea second frame with light emitted from the second and third lightsources at a second time, and wherein luminances of the first wavelengthcomponent of the object obtained from the first and second imagesproduced by capturing the first and second frames are called first andsecond luminances, respectively, and luminances of the second wavelengthcomponent of the object obtained from the first and second images arecalled third and fourth luminances, respectively, the controller isconfigured to obtain information about the distance to the object basedon a ratio of a fifth luminance which is obtained by either choosing onefrom, or synthesizing together, the first and second luminances to asixth luminance which is obtained by either choosing one from, orsynthesizing together, the third and fourth luminances.
 16. Anelectronic device comprising: the motion sensor device of claim 1; and adisplay which changes what to present thereon in response to an object'smotion detected by the motion sensor device.
 17. An integrated circuitfor use in the motion sensor device of claim 1, the integrated circuitcomprising: a timing controller which is connected to the image sensorand the light sources to control timings of exposure and light emission;an extreme value searching section which processes image capturing datato search an area with a relatively high luminance; a coordinate memorywhich stores the coordinates and luminance of the area that has beensearched by the extreme value searching section; and a distancecalculating section which calculates estimated distance informationbased on a luminance ratio by choosing frames that have been shot in thesame area under different conditions from data in the coordinate memory.18. A distance estimating method to be performed by the motion sensordevice of claim 1, the method comprising: searching for an extreme valueby extracting selectively a range with a relatively high light intensityfrom data of captured frames and by calculating its coordinates andlight intensity; calculating a ratio of luminances that have beenextracted in the extreme value searching step and that have beenselected from among luminances of frames shot under differentconditions; and converting the luminance ratio calculated in theluminance ratio calculating step and the coordinates searched for in theextreme value searching step into a distance.
 19. A computer programwhich is defined to make the motion sensor device of claim 1 perform thesteps of: searching for an extreme value by extracting selectively arange with a relatively high light intensity from data of capturedframes and by calculating its coordinates and light intensity;calculating a ratio of luminances that have been extracted in theextreme value searching step and that have been selected from amongluminances of frames shot under different conditions; and converting theluminance ratio calculated in the luminance ratio calculating step andthe coordinates searched for in the extreme value searching step into adistance.